Papers with BioBERT model
Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained Language Models (PTLMs) are typically pretraining on target-domain text, which is expensive in terms of hardware, runtime and CO 2 emissions. |
| Approach: | They propose a faster, CPU-only domainadaptation method that trains Word2Vec on target-domain text and aligns the resulting word vectors with the wordpiece vectors of a general-domain PTLM. |
| Outcome: | The proposed method covers 60% of the BioBERT - BERT F1 delta, 5% of BioBERTS’s CO2 footprint and 2% of its cloud compute cost. |